Deep Learning of ! Human Motion!

Complex human actions are modelled as graphs over basic attributes and their spatial and temporal relationships. The attribute dictionary as well as the graphical structure are learned automatically from training data. y(x,w) = D X i=0 wixi 1 The activity model A set C = {ci, c2, . . . , cC} of C complex activities is considered, where each activity ci is modelled as a Graph Gi = (Vi, Ei), and both, nodes and edges of the graphs, are valued. For convenience, in the following we will drop the index i and consider a single graph for a given activity. The set of nodes V of the graph G is indexed (here denoted by j) and corresponds to occurrences of basic attributes. Each node j is assigned a vector of 4 values {aj, xj, yj, tj} : the attribute type aj and a triplet of spatio-temporal coordinates xj, yj and tj. The edges define pairwise logical spatial or temporal relationships: before, after, overlaps, is included, near, . . .). Examples of graphs for the three activities Leaving a baggage unattended, Telephone conversation, and Handshaking between two people are shown in figure 1. Note, that one node of a model graph can be matched with several consecutive attribute occurrences in the test video. For instance, when the model Telephone conversation (figure 1b) is matched, the node Person will in general be matched to multiple occurrences of a person in the video — as long as the conversation will last. Also, the graphs shown in the figure are only examples, the actual graphs are learned automatically and will in general not correspond to a graph designed by a human. The attribute type variables aj = k may take values k in an alphabet ⇤ = {1, . . . , L}. These values can correspond to fixed (manually designed types), as for instance Person, or automatically learned attributes. Associated to each possible type k is a feature function k(v, x, y, t;⇥k) ! {0, 1} which evaluates whether in the spatio-temporal block centered on (x, y, t) of the video v the attribute is found (= 1) or not (= 0). The parameters (to be learned) of these functions are denoted by ⇥k. Each edge ejk between two nodes j and k is assigned an edge label which may take values in an alphabet ⌥ (before, after, overlaps, is included, near, . . .).). There may be multiple edges between the same pair of nodes. 1 11! Multi-layer Perceptron (MLP)! « Fully-connected » layers! 5.1. Feed-forward Network Functions 229 notation for the two kinds of model. We shall see later how to give a probabilistic interpretation to a neural network. As discussed in Section 3.1, the bias parameters in (5.2) can be absorbed into the set of weight parameters by defining an additional input variable x0 whose value is clamped at x0 = 1, so that (5.2) takes the form aj = D ∑ i=0 w ji xi. (5.8) We can similarly absorb the second-layer biases into the second-layer weights, so that the overall network function becomes yk(x,w) = σ ( M ∑